Evaluating the Interrater Agreement of Process Capability Ratings
نویسندگان
چکیده
The reliability of process assessments has received some study in the recent past, much of it being conducted within the context of the SPICE trials. In this paper we build upon this work by evaluating the reliability of ratings on each of the practices that make up the SPICE capability dimension. The type of reliability that we evaluate is interrater agreement: the agreement amongst independent assessors capability ratings. Interrater agreement was found to be generally high. We also identify one particular practice that exhibits low agreement in its ratings.
منابع مشابه
Cost Implications of Interrater Agreement for Software Process Assessments
Much empirical research has been done recently on evaluating and modeling interrater agreement in software process assessments. Interrater agreement is the extent to which assessors agree in their ratings of software process capabilities when presented with the same evidence and performing their ratings independently. This line of research was based on the premise that lack of interrater agreem...
متن کاملThe reliability of ISO/IEC PDTR 15504 assessments
During phase two of the SPICE trials, the Proposed Draft Technical Report version of ISO/IEC 15504 is being empirically evaluated. This document set is intended to become an international standard for Software Process Assessment. One thread of evaluations being conducted during these trials is the extent of reliability of assessments based on ISO/IEC PDTR 15504. In this paper we present the fir...
متن کامل360’ Ratings: an Analysis of Assumptions and a Research Agenda for Evaluating Their Validity
This article argues that assumptions surrounding 360” ratings should be examined; most notably, the assumptions that different rating sources have relatively unique perspectives on performance and multiple rating sources provide incremental validity over the individual sources. Studies generally support the first assumption, although reasons for interrater disagreement across different organiza...
متن کاملInterrater agreement between visual analysts of single-case data: a meta-analysis.
Visual analysis is the most widely applied method of data interpretation for single-case research as it encompasses multifaceted considerations relevant to evaluating behavior change. However, a previous research synthesis found low levels of interrater agreement between visually analyzed ratings of graphed data across all variables under analysis. The purpose of this meta-analysis was to evalu...
متن کاملThe intra- and interrater reliability of the action research arm test: a practical test of upper extremity function in patients with stroke.
OBJECTIVES To determine the intra- and interrater reliability of the Action Research Arm (ARA) test, to assess its ability to detect a minimal clinically important difference (MCID) of 5.7 points, and to identify less reliable test items. DESIGN Intrarater reliability of the sum scores and of individual items was assessed by comparing (1) the ratings of the laboratory measurements of 20 patie...
متن کامل